Virtual reality 3d eye-inspection by combining images from position-tracked optical visualization modalities

ABSTRACT

A medical visualization apparatus includes two or more imaging devices, two or more robotic arms, multiple magnetic sensors coupled with the imaging devices, and a processor. The two or more imaging devices are configured to acquire images of an organ of a patient. The two or more robotic arms are configured to move the respective imaging devices. The multiple magnetic sensors are configured to output, in response to a magnetic field of a position tracking system, signals indicative of positions and viewing directions of the imaging devices. The processor is configured to estimate a position and a viewing direction of each of the imaging devices based on the signals, and, using the estimated positions and viewing directions, combine the images of the organ acquired by the imaging devices into a virtual reality (VR) image of the organ, and present the VR image to a user on a VR viewer.

FIELD OF THE INVENTION

The present invention relates generally to medical position-trackablerobotic organ-examination systems, and particularly to visual inspectionof an eye using position-trackable robotic systems employing virtualreality.

BACKGROUND OF THE INVENTION

An eye is a complex optical system which collects light from thesurrounding environment, regulates its intensity through a diaphragm,focuses it through an adjustable assembly of lenses to form an image,converts this image into a set of electrical signals, and transmitsthese signals to the brain through complex neural pathways that connectthe eye via the optic nerve to the visual cortex and other areas of thebrain. To this end, the eye consists of a multi layered 3D eyeball madeof various types of tissue, with each tissue having its unique materialcharacteristics and features, including low contrast patterns located atdifferent sections of the eye and are hard to visualize, such as thelens, blood vessels and nerves.

Various techniques to visualize a volume of eye, such as for eye surgeryplanning, were proposed in the patent literature. For example, U.S. Pat.No. 10,517,760 describes systems and methods for aiding a surgeon toperform a surgical procedure on an eye. Exemplary systems include anoptical microscope for the surgeon to view the eye with a microscopeimage during the procedure; an optical coherence tomography (OCT)apparatus configured to perform an OCT scan of a target location in thetarget tissue region during the procedure; and an image processingapparatus configured to generate an augmented image by overlaying an OCTimage of target location and a graphical visual element identifying thelocations, wherein the graphical visual element is registered with themicroscope image to aid the surgeon in advancing a distal end of theelongate probe to the target location.

As another example, U.S. Patent Application Publication 2014/0160264describes an augmented field of view imaging system that includes amicroscope, an image sensor system arranged to receive images of aplurality of fields of view from the microscope as the microscope ismoved across an object being viewed and to provide corresponding imagesignals, an image processing and data storage system configured tocommunicate with the image sensor system to receive the image signalsand to provide augmented image signals, and at least one of an imageinjection system or an image display system configured to communicatewith the image processing and data storage system to receive theaugmented image signals and display an augmented field of view image.The image processing and data storage system is configured to track theplurality of fields of view in real time and register the plurality offields of view to calculate a mosaic image. The augmented image signalsfrom the image processing and data storage system provide the augmentedimage such that a live field of view from the microscope is compositedwith the mosaic image.

SUMMARY OF THE INVENTION

An embodiment of the present invention that is described hereinafterprovides a medical visualization apparatus including two or more imagingdevices, two or more robotic arms, multiple magnetic sensors, and aprocessor. The two or more imaging devices are configured to acquireimages of an organ of a patient. The two or more robotic arms areconfigured to move the respective imaging devices. The multiple magneticsensors, which are coupled with the two or more imaging devices, areconfigured to output, in response to a magnetic field of a positiontracking system, signals indicative of positions and viewing directionsof the two or more imaging devices. The processor is configured to: (a)estimate a position and a viewing direction of each of the imagingdevices based on the signals, and b) using the estimated position andviewing direction of each of the imaging devices, combine the images ofthe organ acquired by the two or more imaging devices into a virtualreality image of the organ, and present the virtual reality image to auser on a virtual reality viewing device.

In some embodiments, at least one of the two or more imaging devicesincludes one or more selectable wavelength filters.

In some embodiments, each of the two or more imaging devices include atleast one of a 3D microscope, a thermal camera, and an OCT device.

In an embodiment, the two or more imaging devices include at least twomicroscope objectives.

In some embodiments, the processor is configured to move at least one ofthe two or more robotic arms according to a user request specifying agazing direction.

In other embodiments, the processor is configured to combine the imagesindependently of any coordinate-system registration between the two ormore imaging devices.

In an embodiment, at least one of the multiple magnetic sensors isconfigured to output, in response to the magnetic field, one or moresignals indicative of a roll angle of an imaging device about theestimated viewing direction of at least one of the two or more imagingdevices, and wherein the processor is further configured to estimate theroll angle based on the one or more signals.

In another embodiment, wherein at least one of the two or more imagingdevices include a light polarizer, and wherein the processor isconfigured to adjust the light polarizer by adjusting the roll angle.

There is additionally provides, in accordance with another embodiment ofthe present invention, a medical visualization method includingacquiring images of an organ of a patient using two or more imagingdevices. The imaging devices are moved using two or more respectiverobotic arms. In response to a magnetic field of a position trackingsystem, signals are generated, that are indicative of positions andviewing directions of the two or more imaging devices, using multiplemagnetic sensors which are coupled with the two or more imaging devices.A position and a viewing direction of each of the imaging devices areestimated based on the signals. Using the estimated position and viewingdirection of each of the imaging devices, the images of the organacquired by the two or more imaging devices are combined into a virtualreality image of the organ, and the virtual reality image is presentedto a user on a virtual reality viewing device.

The present invention will be more fully understood from the followingdetailed description of the embodiments thereof, taken together with thedrawings in which:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic, pictorial view of an eye-inspection systemcomprising position-trackable robot-mounted eye-imaging devices andvirtual-reality eyeglasses, in accordance with an embodiment of thepresent invention;

FIG. 2 is a schematic, pictorial view of the position-trackablerobot-mounted eye-imaging devices of the eye inspection system of FIG.1, in accordance with an embodiment of the present invention;

FIG. 3 is a schematic, pictorial view of a position-trackable roboticeye inspection system, in accordance with another embodiment of thepresent invention; and

FIG. 4 is a flow chart schematically illustrating a method for using theposition-trackable robot eye inspection system of FIG. 1, in accordancewith an embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS Overview

Embodiments of the present invention that are described hereinaftercombine images of an organ (e.g., an eye) being diagnosed and/or treated(e.g., surgically), that are taken from multiple optical imaging devicesof one or more imaging modalities, that can be mounted on robotic arms,such as OCT images and 3D digital microscope images. The position andthe viewing direction of each of the optical imaging devices are trackedwith a magnetic tracking system, by tracking multiple magnetic sensorscoupled to the imaging devices, and the processor combines the producedimages into a multi-layered 3D image of the eye. The 3D image is thentransmitted to a virtual reality viewing device, such as virtual realityeyeglasses, worn by the physician.

The physician can select the gaze direction to view the eye, forinstance by using an add-on for the finger to act as a virtual pointer.Using the tracked position and direction, a processor aligns all imagesand compensates for motion without a need for careful registration ofdifferent frames of references of the different modalities that may bemoved during the process intentionally (by the robotic arms) orunintentionally (for example by actions of personnel in a sophisticatedsurgical theater setting).

In some embodiments, at least some of the optical imaging devices, suchas the 3D digital microscope, are held by robot arms which are trackedby the magnetic tracking system. The robot arm is manipulated so thatmultiple images (e.g., microscope images and/or OCT images) are acquiredfrom different viewing directions. The physician is able to select aparticular viewing direction, for example of the 3D microscope, and theimages are then presented to the physician in a virtual reality deviceby, for example, toggling the two images between the physician's leftand right eyes.

The optical imaging modalities may have numerous dissimilarcharacteristics, such as, for example, different focal lengths,different fields of view, and/or different operating speeds and opticalwavelengths. At least some of the optical imaging devices may comprise,for example miniature cameras such as a fundus camera. In someembodiments, at least some of the optical imaging devices comprise oneor more filters, which allow the physician to select a wavelength range,e.g. the visible spectrum, near- or far-infrared, or thermalwavelengths, for viewing the eye.

In some embodiments, at least one of the multiple magnetic sensors isconfigured to output, in response to a magnetic field of the positiontracking system, signals indicative of a roll angle of an imagingdevice, and the processor is configured to, based on the signals,estimate the roll angle about the estimated direction. In thisembodiment, two or more imaging devices may comprise one or more lightpolarizers that the processor adjusts by adjusting the roll angle.

The optical imaging devices (e.g., cameras) are distributed such thattheir images cover a substantial portion of the eye. The optical imagingdevices may be standalone systems or may be attached to other systemelements, such as to a respective objective of a microscope.

The OCT images are tomographic images, i.e., slices, that are orthogonalto the optical axis of an OCT assembly. The physician can select fromwhich part of the optical axis the OCT image is to be used in thevirtual reality device.

In some embodiments, a microscope objective is mounted on a robotic armwith six degrees of freedom, which is tracked with a magnetic trackingsystem. The image of the object acquired by the objective is transferredto virtual reality eyeglasses worn by a user, who is then able tooperate the robotic arm so as to select a direction of observation ofthe objective. The combination of the robotic microscope coupled tovirtual reality eyeglasses gives much greater freedom of viewingdirection than is possible with a standard microscope.

A further use of the robotic arm-mounted microscope may be to build a 3Dimage of the viewed eye. In this case the objective is assumed to have asmall depth of field. The objective is moved along its optical axis, andis used to capture images of elements of the object that are in sharpfocus. These images are used to build a 3D image of the object.

In another embodiment, two microscopes mounted on robotic arms are used(possibly with one or more additional optical imaging modalities), eachrobotic arm having six degrees of freedom, to which a microscopeobjective is attached. The positions, viewing directions, and,optionally, roll-angles, of the objectives are magnetically trackedusing a magnetic tracking system. Each image of the viewed object,acquired by a respective objective, is transferred to virtual realityeyeglasses worn by a user, and the separate images are presented to theuser's left and right eye. The user views the separate images as in astandard digital 3D microscope.

However, the robotic arms of the invention enable the user to view theobject from a much wider range of viewing directions compared to thestandard 3D digital microscope. In addition, the user is able to changethe separation of the objectives at will, so as to enhance or reduce the3D effect.

Using the disclosed virtual 3D optical imaging techniques that includeaugmenting images from multiple optical imaging modalities that areposition tracked into a 3D virtual reality image, may allow theperformance of eye surgeries more easily, more effectively, and withfewer hazards.

System Description

FIG. 1 is a schematic, pictorial view of an eye-inspection system 10comprising position-trackable robot-mounted eye imaging devices (48, 58,68) and virtual reality eyeglasses 133, in accordance with an embodimentof the present invention.

System 10 has robotic arms 44, 55 and 66 mounted on a fixed base 102(e.g., suspended from a ceiling). Imaging device 48 may have a camerawith an optical imaging axis 144 that is coupled with robotic arm 44;imaging device 58 may include an OCT, including an objective with anoptical imaging axis 155 and a camera, that is coupled to robotic arm55; and imaging device 68 may have a 3D digital microscope with anoptical imaging axis 166 with a variable focus length on axis 166 thatis coupled with robotic arm 66.

The robotic arms are controlled by a processor 38 that may varydirections 144/155/166 and/or depth of imaging along direction144/155/166, according to a gaze direction selected by the physician toview the eye, for instance by using a pointing device such as a mouse ora trackball of a user interface 40, or by moving a finger add-on (notshown) to act as a virtual pointer.

As inset 25 shows, directions 144/155/166 are aligned so as to providean unobstructed optical path for each of the imaging devices (48/58/68)to view a lens 18 of an eye 20 of a patient 19.

In an embodiment, system 10 is equipped with robotic arms (44, 55, 66)having six degrees of freedom, though the number of degrees of freedommay vary with design, typically with a minimum of two or three (e.g., topoint at a solid angle direction (2), and vary depth of focus (1)).

During eye inspection, one or more of robotic arms 44/55/66 move theimaging devices (48/58/68) according to commands from processor 38communicated via cables 31/43/46 running between a console 28 and base102, where cable 31 is further used to convey signals from imagingdevice 48 having a thermal camera to processor 38. Cable 43 is furtherused to convey signals from the camera behind imaging device 58 having acamera objective to OCT subsystem 24, and cable 46 is further used toconvey signals from imaging device 68 having a 3D digital microscope toa 3D digital microscope subsystem 26.

In the shown embodiment, system 10 comprises a magnetic-sensingsubsystem 101 to estimate positions and directions of the imagingdevices (48/58/68). To this end, patient 19 is placed in a magneticfield generated by a pad containing magnetic field generator coils 120,which are driven by unit 30 via a cable 33. The magnetic fieldsgenerated by coils 120 generate position signals in magnetic sensors110, each coupled to imaging devices (48/58/68). The signals from eachsensor 110 are then provided, as corresponding electrical inputs, toprocessor 38 to calculate the separate position and direction of each ofimaging devices (48/58/68).

The method of position sensing using external magnetic fields andmagnetic sensor is implemented in various medical applications, forexample, in the CARTO™ system, produced by Biosense-Webster, and isdescribed in detail in U.S. Pat. Nos. 5,391,199, 6,690,963, 6,484,118,6,239,724, 6,618,612 and 6,332,089, in PCT Patent Publication WO96/05768, and in U.S. Patent Application Publications 2002/0065455 A1,2003/0120150 A1 and 2004/0068178 A1, whose disclosures are allincorporated herein by reference.

In the shown embodiment, imaging device 48 having a thermal cameracaptures a thermal image of lens 18 in real time. The captured image 69is displayed on a display 36. Beyond its possible use in diagnostics,the displayed thermal image enables physician 15 to monitor temperatureand prevent thermal hazard to eye 20 during surgery (e.g., laser orfocused ultrasound, neither of which are shown).

Processor 38 presents other results of a diagnostic and/or therapeuticprocedure on display 36. As noted above, processor 38 may receiveuser-based commands via a user interface 40. User interface 40 may becombined with a touchscreen graphical user interface of display 36.

Some or all of the functions of processor 38 may be combined in a singlephysical component or, alternatively, implemented using multiplephysical components. These physical components may comprise hard-wiredor programmable devices, or a combination of the two. In someembodiments, at least some of the functions of processor 38 may becarried out by suitable software stored in a memory 35 (as shown in FIG.1). This software may be downloaded to a device in electronic form, overa network, for example. Alternatively, or additionally, the software maybe stored in tangible, non-transitory computer-readable storage media,such as optical, magnetic, or electronic memory.

The apparatus shown in FIG. 1 may include further elements, which areomitted for clarity of presentation. For example, physician 15 may holda control handle which can, for example, command processor 38. Physician15 may use surgical tools and/or apply medications, which are also notshown in order to maintain clarity and simplicity of presentation.

Virtual Reality 3D Eye Inspection by Combining Images fromPosition-Tracked Optical Imaging Modalities

FIG. 2 is a schematic, pictorial view of the position-trackable roboticarms 44/55/66 mounted eye imaging devices 48/58/68 of eye inspectionsystem 10 of FIG. 1, in accordance with an embodiment of the presentinvention.

As seen, imaging devices 48/58/68 are arranged in space to haveunobstructed views 144/155/166 of eye 20, and the imaging devices can bemaneuvered by respective robotic arms 44/55/66 so that the arrangementof the different acquisitions (positions and directions) can be adjustedby processor 38, as directed by physician 15 by selecting a gazedirection using a gaze direction selector 215 to view the eye, togenerate a combined 3D image 210 of eye 20 that physician 15 views onvirtual reality eyeglasses 233. Moreover, combined 3D image 210 maypresent the physician information related to a wavelength range selectedvia a wavelength filter/selector 225 for viewing the eye, e.g. thevisible spectrum, near- or far-infrared, or thermal wavelengths.

To this end, using the tracked position and viewing direction of imagingdevices 48/58/68, obtained by magnetic tracking system 101 using sensors110, processor 38 aligns all images (thermal images 204, OCT images 205,and 3D camera images 206). Processor 38 compensates for motion ofimaging devices 48/58/68 without a need for registration of differentframes of reference of the different modalities (devices), that may bemoved during the procedure, either intentionally (by the robotic arms)or unintentionally, for example, by actions of personnel in asophisticated surgical theater setting.

In the shown embodiment, an image combiner module 202 of processor 38combines thermal images 204, OCT images 205 and 3D microscope cameraimages 206 into 3D image 210, the microscope having a variable focallength to enable 3D image creation by scanning a moving object planealong direction 166 (the plane normal to direction 166).

Physician 15 can receive more information on virtual reality eyeglasses233, by, for example, varying wavelength filter types of the 3D digitalmicroscope of imaging device 68 and/or the OCT device of imaging device58 and/or the thermal camera of imaging device 48. Physician 15 canchange filters and/or gazing direction using controls specified inFIG. 1. Using additional controls physician 15 may choose to switchbetween combined image 210 to images from any of one or more of themodalities.

The example unit shown in FIG. 2 is chosen purely for the sake ofconceptual clarity. For example, combined image 210 may be generatedfrom more than three imaging modalities.

Virtual Reality 3D Microscope

FIG. 3 is a schematic, pictorial view of a position-trackable roboticeye-inspection system 300, in accordance with another embodiment of thepresent invention. System 300 comprises two focus cameras (303, 305)each including a microscope objective coupled to respective robotic arms(302, 304) that are coupled with a stationary rail 343. The robotic armshave a separation 319 between them, which can be adjusted, for example,by using a motorized assembly on a rail (not shown). The user is able tochange the separation of the objectives in order to receive an enhancedor reduced 3D effect at will, depending on the angle 322 between theviewing directions (333, 355) of the objectives.

The two robotic arms each have six degrees of freedom of motion, withthe positions and viewing directions of the objectives magneticallytracked using a magnetic tracking system comprising magnetic fieldgenerator coils 320 similar to coils 120 of FIG. 1, and magnetic sensors310 similar to sensors 110 of FIG. 1.

Each image of the viewed object, acquired by a respective objective, istransferred to virtual reality eyeglasses 313 worn by physician 15, andthe separate images are presented to the left and right eye of thephysician. The physician views the separate images as in a standard 3Dmicroscope. The robotic arms, when combined with the magnetic tracking,enable the user to view the object (e.g., an eye 20) from a much widerrange of viewing directions, depending for example, on selecting a gazedirection via a gaze direction selector 340 (e.g., a direction “into thepage” in FIG. 3) to view the eye, compared to the standard microscope.

Method for Using the Position-Trackable Robot Eye Inspection System

FIG. 4 is a flow chart schematically illustrating a method for using theposition-trackable robot eye inspection system of FIG. 1, in accordancewith an embodiment of the present invention. The algorithm, according tothe presented embodiment, carries out a process that begins withphysician 15 operating apparatus 10 (e.g., processor 38) to starttracking the positions and viewing directions of 3D digital microscopeof imaging device 68, OCT device of imaging device 58 and thermal cameraof imaging device 48, at a magnetic tracking initiation step 402.

Physician 15 then wears virtual reality eyeglasses and verifies thepresence of an image screen, at a VR preparatory step 404.

Next, at a setting step 406, the physician selects a gazing directionand wavelength filters to view a combined image of the organ.

At a next 3D view setting step, the physician uses controls, such asfrom user interface 40, to adjust a depth of focus of 3D microscope ofimaging device 68 for a best 3D view the organ.

Finally, using the controls, the physician toggles between views, suchas between the combined 3D image and one of the images (e.g., of the 3Dmicroscope) at a view toggling step 410.

The example flow chart shown in FIG. 4 is chosen purely for the sake ofconceptual clarity. For example, additional inspection steps, such asnoninvasive intra-ocular pressure measurement, may also be performed.

It will be appreciated that the embodiments described above are cited byway of example, and that the present invention is not limited to whathas been particularly shown and described hereinabove. Rather, the scopeof the present invention includes both combinations and sub-combinationsof the various features described hereinabove, as well as variations andmodifications thereof which would occur to persons skilled in the artupon reading the foregoing description and which are not disclosed inthe prior art. Documents incorporated by reference in the present patentapplication are to be considered an integral part of the applicationexcept that to the extent any terms are defined in these incorporateddocuments in a manner that conflicts with the definitions madeexplicitly or implicitly in the present specification, only thedefinitions in the present specification should be considered.

1. A medical visualization apparatus, comprising: two or more imagingdevices that are configured to acquire images of an organ of a patient;two or more robotic arms configured to move the respective imagingdevices; multiple magnetic sensors, which are coupled with the two ormore imaging devices and are configured to output, in response to amagnetic field of a position tracking system, signals indicative ofpositions and viewing directions of the two or more imaging devices; anda processor, which is configured to: estimate a position and a viewingdirection of each of the imaging devices based on the signals; and usingthe estimated position and viewing direction of each of the imagingdevices, combine the images of the organ acquired by the two or moreimaging devices into a virtual reality image of the organ, and presentthe virtual reality image to a user on a virtual reality viewing device.2. The apparatus according to claim 1, wherein at least one of the twoor more imaging devices comprises one or more selectable wavelengthfilters.
 3. The apparatus according to claim 1, wherein each of the twoor more imaging devices comprise at least one of a 3D microscope, athermal camera, and an OCT device.
 4. The apparatus according to claim1, wherein the two or more imaging devices comprise at least twomicroscope objectives.
 5. The apparatus according to claim 1, whereinthe processor is configured to move at least one of the two or morerobotic arms according to a user request specifying a gazing direction.6. The apparatus according to claim 1, wherein the processor isconfigured to combine the images independently of any coordinate-systemregistration between the two or more imaging devices.
 7. The apparatusaccording to claim 1, wherein at least one of the multiple magneticsensors is configured to output, in response to the magnetic field, oneor more signals indicative of a roll angle of an imaging device aboutthe estimated viewing direction of at least one of the two or moreimaging devices, and wherein the processor is further configured toestimate the roll angle based on the one or more signals.
 8. Theapparatus according to claim 7, wherein at least one of the two or moreimaging devices comprise a light polarizer, and wherein the processor isconfigured to adjust the light polarizer by adjusting the roll angle. 9.A medical visualization method, comprising: acquiring images of an organof a patient using two or more imaging devices; moving the imagingdevices using two or more respective robotic arms; generating, inresponse to a magnetic field of a position tracking system, signalsindicative of positions and viewing directions of the two or moreimaging devices using multiple magnetic sensors which are coupled withthe two or more imaging devices; estimating a position and a viewingdirection of each of the imaging devices based on the signals; and usingthe estimated position and viewing direction of each of the imagingdevices, combining the images of the organ acquired by the two or moreimaging devices into a virtual reality image of the organ, andpresenting the virtual reality image to a user on a virtual realityviewing device.
 10. The method according to claim 9, wherein acquiringthe images comprises applying one or more selectable wavelength filtersin at least one of the imaging devices.
 11. The method according toclaim 9, wherein each of the two or more imaging devices comprise atleast one of a 3D microscope, a thermal camera, and an OCT device. 12.The method according to claim 9, wherein the two or more imaging devicescomprise at least two microscope objectives.
 13. The method according toclaim 9, wherein moving the imaging devices comprises moving the imagingdevices according to a user request specifying a gazing direction. 14.The method according to claim 9, wherein combining the images comprisescombining the images independently of any coordinate-system registrationbetween the imaging devices.
 15. The method according to claim 9,further comprising generating from at least one of the magnetic sensors,in response to the magnetic field, one or more signals indicative of aroll angle of at least one of the two or more imaging devices about theestimated viewing direction of the at least one of the two or moreimaging devices, and estimating the roll angle based on the one or moresignals.
 16. The method according to claim 15, wherein at least one ofthe two or more imaging devices further comprise one or more lightpolarizers; and further comprising adjusting the roll angle of at leastone of the two or more imaging devices.